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Abstract 


The  project  has  been  concerned  with  developing  new  numerical 
techniques  to  solve  large  scale  linear  and  nonlinear  programming 
problems.  Early  work  focused  on  sequential  quadratic  programming 
techniques  for  nonlinear  programming.  Subsequently,  all  work  was 
focused  on  interior  point  methods  for  large  scale  linear  and 
nonlinear  programming.  Initially,  the  focus  of  the  research  was 
on  both  dual-affine  and  primal-dual  algorithm  for  linear 
programming.  Substantial  computational  experience  demonstrated 
the  superiority  of  the  primal-dual  methods,  and  subsequent 
research  focused  on  improving  the  efficiency  of  these  methods, 
both  by  adding  higher  order  methods  via  predictor-corrector 
techniques  and  by  improving  the  linear  algebra  to  take  advantage 
of  both  sparsity  and  machine  architecture.  Most  recently, 
research  has  focused  on  large  scale  quadratic  programming.  A 
primal-dual  predictor-corrector  method  has  been  devised  and  shown 
to  be  very  promising  computationally  for  problems  with  diagonal 
or  sparsely  structured  Hessian  matrices.  For  problems  with  dense 
Hessians,  a  pure  primal  conjugate  projected  gradient  algorithm 
shows  promise  on  small  problems.  It  remains  to  be  tested  on 
large-scale  problems. 
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Summary  of  Research  Accomplishments 


The  initial  research  undertaken  under  the  grant  was  a 
numerical  study  of  various  topics  in  sequential  quadratic 
programming.  The  topics  studied  included  Lagrange  multiplier 
estimates,  choice  of  merit  function,  choice  of  search  direction, 
and  choice  of  penalty  parameter.  Extensive  testing  determined  a 
stable  algorithm  for  the  standard  test  set,  and  indicated  the 
need  for  further  research  in  multiplier  estimates,  research  which 
will  be  undertaken  in  the  next  year. 

All  other  research  throughout  the  duration  of  the  grant  was 
concerned  with  interior  point  methods  for  linear  and  nonlinear 
programming.  Initially,  research  into  developing  a  viable 
computational  algorithm  took  two  separate  directions,  a  full 
scale  implementation  of  a  dual  affine  algorithm  and  a  prototype 
implementation  of  a  primal-dual  algorithm.  The  dual-affine 
implementation  served  as  an  interesting  platform  for  devising 
algorithms  for  bounded  variables,  eliminating  rows  and  columns, 
testing  starting  points,  and  for  examining  the  importance  of 
centering. 

The  prototype  studies  of  the  primal-dual  algorithm 
demonstrated  that  the  method  had  great  computational  promise,  and 
was  far  more  stable  than  the  dual  affine  algorithm.  Analysis 
demonstrated  that  this  stability  was  largely  due  to  the  automatic 
centering  of  the  algorithm.  In  view  of  this,  implementation  of  a 
full  scale  primal-dual  algorithm  was  undertaken.  This 
implementation  proved  so  satisfactory  that  all  further  research 
for  linear  programming  concentrated  on  this  algorithm. 

Work  on  the  prototype  algorithm  had  demonstrated  how  to 
incorporate  bounded  variables  and  ranges  into  the  algorithm. 
Early  work  on  the  full  implementation  of  the  original  algorithm 
focused  on  improved  algorithms  for  choice  of  the  centering 
parameter,  better  initial  estimates  to  the  solution,  and  better 
ways  to  find  feasible  solutions.  A  major  breakthrough  in 
handling  infeasibility  occurred  when  Newton's  method  was  applied 
directly  to  the  first  order  necessary  conditions  from  an 
infeasible  starting  point.  This  method  has  proved  highly 
successful  in  practice,  and  remains  the  standard  way  of  handling 
infeasibility. 

A  great  deal  of  research  dealt  with  the  problem  of  special 
handling  of  dense  columns  in  the  constraint  matrix.  These 
columns  can  make  the  Cholesky  factorization  of  the  matrix  of 
normal  equations  unacceptably  dense,  but  this  problem  can  be 
overcome  by  careful  special  handling.  While  reasonable 
techniques  to  handle  dense  columns  have  been  implemented,  the 
problem  remains  one  for  further  study. 


The  next  major  advance  in  the  linear  programming  algorithm 
came  with  the  implementation  of  higher-order  methods  via  the 
predictor-corrector  technique.  In  addition  to  implementing  the 
predictor-corrector  method,  which  reduced  run  time  by 
approximately  40%,  research  concentrated  on  analyzing  the  method 
and  investigating  higher-order  predictor-corrector  methods. 
While  higher-order  methods  proved  theoretically  interesting,  a 
single  corrector  term  has  proved  computationally  the  most 
efficient  algorithm. 

Further  speedups  of  the  algorithm  came  from  improving  the 
numerical  linear  algebra.  Incorporation  of  supernodes  and  loop 
unrolling  improved  speed  up  to  40%  on  large  problems.  Further 
tailoring  the  choice  of  matrix  ordering  algorithm  and  Cholesky 
factorization  to  specific  architectures  significantly  improved 
solution  times.  Research  continues  on  improvements  in  this  area, 
particularly  on  matrix  orderings. 

More  recent  research  has  concentrated  on  applying  interior 
point  methods  to  nonlinear  programming  problems.  Initial 
research  concentrated  on  applying  straight  primal-dual  and 
primal-dual  predictor-corrector  algorithms  to  separable  quadratic 
programming  problems.  Computational  results  even  extremely 
encouraging,  and  have  led  to  further  research  in  two  areas; 
nonseparable  problems  with  sparse  Hessian  matrices  and 
nonseparable  large-scale  problems  with  dense  Hessian  matrices. 
Primal-dual  predictor-corrector  algorithms  have  been  successfully 
implemented  for  sparse  Hessians.  However,  if  the  Hessian  is 
large  and  dense,  factorizing  the  matrix  can  become  highly 
inefficient. 

To  attack  problems  with  large  dense  Hessians,  but  hopefully 
multiple  clustered  eigenvalues,  a  primal  algorithm  based  on 
conjugate  projected  gradients  has  been  devised  and  tested  in  a 
prototype  version.  Computational  results  are  sufficiently 
encouraging  that  work  is  currently  underway  to  develop  a  full 
blown  code  for  large  problems. 

Finally,  when  acceptable  algorithms  for  quadratic 
programming  have  been  fully  developed,  these  will  be  incorporated 
as  the  quadratic  programming  algorithm  in  a  sequential  quadratic 
programming  method  for  nonlinear  programming.  Thus  during  the 
next  phase  of  the  research,  the  topic  will  come  full  circle  back 
to  the  original  study  on  sequential  quadratic  programming,  but 
with  the  capability  of  solving  far  larger  problems  than  when  the 
research  was  initiated. 
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Abstracts 


"The  Primal-Dual  Interior  Point  Method  on  the  Cray  Supercomputer" 

This  paper  discusses  computational  experience  on  the  Cray  Y- 
MP  supercomputer  with  the  primal-dual  interior  point  code  OBI. 
Results  are  given  for  a  variety  of  medium  to  large  size  models 
with  comparisons  to  workstation  implementations  and  published 
results  for  the  KORBX  System.  Empirical  results  on  the  growth  of 
the  iteration  count  with  the  number  of  variables  are  given  and 
areas  for  future  research  are  discussed. 


"On  Implementing  Mehrotra's  Predictor-Corrector  Interior  Point 
Method  for  Linear  Programming" 

Mehrotra  recently  described  a  predictor-corrector  variant  of 
the  primal-dual  interior  point  algorithm  for  linear  programming. 
This  paper  describes  a  full  implementation  of  this  algorithm, 
with  extensions  for  solving  problems  with  free  variables  and 
problems  with  bounds  on  primal  variables.  Computational  results 
on  the  NETLIB  test  set  are  given  to  show  that  this  new  method 
almost  always  improves  the  performance  of  the  primal-dual 
algorithm  and  that  the  improvement  increases  dramatically  as  the 
size  and  complexity  of  the  problem  increases.  A  numerical 
instability  in  using  Schur  complements  to  remove  dense  columns  is 
identified,  and  a  numerical  remedy  is  given. 


"Higher  Order  Predictor-Corrector  Interior  Point  Methods  with 
Application  to  Quadratic  Objectives" 

In  this  paper,  we  explore  the  full  utility  of  Mehrotra's 
predictor-corrector  method  in  the  context  of  linear  and  convex 
quadratic  programs.  We  describe  a  procedure  for  doing  multiple 
corrections  at  each  iteration  and  implement  it  within  the 
framework  of  several  strategies  for  determining  the  number  of 
corrections  in  a  given  iteration.  The  results  indicate  the 
iteration  counts  can  be  significantly  reduced  by  allowing  higher 
order  corrections  but  at  the  cost  of  extra  work  per  iteration. 
The  procedure  is  shown  to  be  a  level-m  composite  Newton  interior 
point  method  where  m  is  the  number  of  corrections  performed  in  an 
iteration. 


"The  Interaction  of  Algorithms  and  Architectures  for  Interior 
Point  Methods" 

This  paper  examines  the  minimum  local  fill-in  and  multiple 
minimum  degree  reordering  strategies  for  sparse  positive  definite 
symmetric  systems  of  equations  arising  from  interior  point 
methods  for  linear  programming.  The  performance  of  each  ordering 
strategy  is  studied  with  both  leftward  and  rightward  looking 
sparse  Cholesky  decompositions  utilizing  supernode  and  loop 
unrolling  techniques.  The  various  algorithms  are  tested  on  a 
DECstation  3100,  ..a  IBM  RISC/System  6000  Powerstation  530,  and 
CRAY  Y-MP.  Computational  results  are  given  to  show  that  the 
correct  choice  of  algorithm  is  hardware  dependent. 


"Separable  Quadratic  Programming  via  a  Primal-Dual  Point  Method 
and  its  use  in  a  Sequential  Procedure" 

This  paper  extends  a  primal-dual  interior  point  procedure 
for  linear  programs  to  the  case  of  convex  separable  cfuadratic 
objectives.  Included  are  efficient  procedures  for;  attaining 
primal  and  dual  feasibility,  variable  upper  bounding,  and  free 
variables.  A  sequential  procedure  that  invokes  the  quadratic 
solver  is  proposed  and  implemented  for  solving  linearly 
constrained  convex  separable  nonlinear  programs.  Computational 
results  are  provided  for  several  large  test  cases  from  stochastic 
programming.  The  proposed  methods  compare  favorably  with  MINOS, 
especially  for  the  larger  examples.  The  nonlinear  programs  range 
in  size  up  to  8700  constraints  and  22000  variables. 


"Very  Large-Scale  Linear  Programming;  A  Case  Study  in  Combining 
Interior  Point  and  Simplex  Methods" 

Experience  with  solving  a  12,753,313  variable  linear  program 
is  described.  This  problem  is  the  linear  programming  relaxation 
of  a  set  partitioning  problem  arising  from  an  airline  crew 
scheduling  application.  A  scheme  is  described  that  requires 
successive  solutions  of  small  subproblems,  yielding  a  procedure 
that  has  little  growth  in  solution  time  in  terms  of  the  number  of 
variables.  Experience  using  the  simplex  method  as  implemented  in 
CPLEX,  an  interior  point  method  as  implemented  in  OBI,  and  a 
hybrid  interior  point/simplex  approach  is  reported.  The 
resulting  procedure  illustrates  the  power  of  an  interior 
point/simplex  combination  for  solving  very  large-scale  linear 
programs . 


"Numerical  Experience  with  Sequential  Quadratic  Programming 
Algorithms  for  Equality  Constrained  Nonlinear  Programming" 

Computational  experience  is  given  for  a  sequential  quadratic 
programming  algorithm  when  Lagrange  multiplier  estimates,  Hessian 
approximations,  and  merit  functions  are  varied  to  test  for 
computational  efficiency.  Indications  of  areas  for  further 
research  are  given. 


"Computing  Karmarkar  Projections  Quickly" 

The  paper  presents  a  numerical  method  for  computing  the 
projections  for  Karmarkar 's  new  algorithm  for  linear  programming. 
The  method  is  simple  to  implement,  fully  exploits  sparsity,  and 
appears  in  limited  experimentation  to  have  good  stability 
properties.  Preliminary  numerical  experience  indicates  that  the 
method  promises  advantages  over  methods  that  refactor  a  matrix  at 
every  iteration. 


"A  Unified  View  of  Interior  Point  Methods  for  Linear  Programming" 

The  paper  shows  how  various  interior  point  methods  for 
linear  programming  may  all  be  derived  from  logarithmic  barrier 
methods.  These  methods  include  primal  and  dual  projective 
methods,  affine  methods,  and  methods  based  on  the  method  of 
centers.  In  particular,  the  paper  demonstrates  that  Karmarkar 's 
algorithm  is  equivalent  to  a  classical  logarithmic  barrier  method 
applied  to  a  problem  in  standard  form. 


"Implementation  of  a  Daal  Affine  Interior  Point  Algorithm  for 
Linear  Programming" 

The  dual  affine  interior  point  method  is  extended  to  handle 
variables  with  simple  upper  bounds  as  well  as  free  variables. 
During  execution,  variables  which  appear  to  be  going  to  zero  are 
fixed  at  zero,  and  rows  with  slack  variables  bounded  away  from 
zero  are  removed.  A  variant  of  the  big-M  artificial  variable 
method  to  attain  feasibility  is  derived.  The  simplex  method  is 
used  to  recover  an  optimal  basis  upon  completion  of  the 
algorithm,  and  the  effects  of  scaling  are  discussed. 
Computational  experience  on  a  variety  of  problems  is  presented. 


"An  Implementation  of  a  Primal-Dual  Interior  Point  Method  for 
Linear  Programming" 

The  purpose  of  this  paper  is  to  describe  in  detail  an 
implementation  of  a  primal-dual  interior  point  method  for  solving 
linear  programming  problems.  Preliminary  computational  results 
indicate  that  this  implementation  compares  favorably  with  a 
comparable  implementation  of  a  dual  affine  interior  point  method, 
and  with  MINOS  5.0,  a  state-of-the-art  implementation  of  the 
simplex  method. 


"Further  Development  of  a  Primal-Dual  Interior  Point  Method" 

The  paper  continues  the  development  of  a  primal-dual 
interior  point  algorithm  for  linear  programming.  The  topics 
studied  include  simple  bounds  on  primal  variables,  incorporation 
of  Lustig's  phase  1  algorithm  and  the  use  of  Schur  complements  to 
handle  dense  columns.  Extensive  numerical  results  demonstrate 
the  efficiency  of  the  resulting  algorithm. 


"Computational  Experience  with  a  Primal-Dual  Interior  Point 
Method  for  Linear  Programming" 

A  new  comprehensive  implementation  of  a  primal-dual 
algorithm  for  linear  programming  is  described.  It  allows  for 
easy  handling  of  simple  bounds  on  the  primal  variables  and 
incorporates  free  variables,  which  have  not  previously  been 
included  in  a  primal-dual  implementation.  We  discuss  in  detail  a 
variety  of  computational  issues  concerning  the  primal-dual 
implementation  and  barrier  methods  for  linear  programming  in 
general.  We  show  that,  in  a  certain  way,  Lustig's  method  for 
obtaining  feasibility  is  equivalent  to  Newton's  method.  This 
demonstrates  that  the  method  is  in  some  sense  the  natural  way  to 
reduce  infeasibility.  The  role  of  the  barrier  parameter  in 
computational  practice  is  studied  in  detail.  Numerical  results 
are  given  for  the  entire  expanded  NETLIB  test  set  for  the  basic 
algorithm  and  its  variants,  as  well  as  version  5.3  of  MINOS. 


"Starting  and  Restarting  the  Primal-Dual  Interior  Point  Method" 

We  first  consider  the  problem  of  choosing  a  starting  point 
for  an  interior  point  algorithm  for  linear  programming  when  no 
starting  point  is  given.  It  is  shown  that  incorporation  of 
bounds  and  reflection  of  negative  values  in  an  analytic  formula 
for  estimating  the  starting  point  can  improve  performance  over 


earlier  methods.  We  than  consider  the  problem  of  restarting  the 
algorithm  from  a  prior  optimal  solution  in  order  to  solve  a 
perturbed  problem.  We  illustrate  how  the  previously  optimal 
solution  must  be  perturbed  in  order  to  be  able  to  make  adequate 
progress  toward  a  new  optimal  point.  Preliminary  numerical 
results  indicate  that  convergence  to  a  new  optimum  is  much  faster 
from  the  perturbed  previous  optimum  than  from  a  cold  start. 


"An  Interior  Point  Method  for  Quadratic  Programs  Based  on 
Conjugate  Projected  Gradients” 

We  propose  an  interior  point  method  for  large-scale  convex 
quadratic  programming  where  no  assumptions  are  made  about  the 
sparsity  structure  of  the  quadratic  coefficient  matrix  Q.  The 
interior  point  method  we  describe  is  a  doubly  iterative  algorithm 
that  invokes  a  conjugate  projected  gradient  procedure  to  obtain 
the  search  direction.  The  effect  is  that  Q  appears  in  a 
conjugate  direction  routine  rather  than  in  a  matrix 
factorization.  By  doing  this,  the  matrices  to  be  factored  have 
the  same  nonzero  structure  as  those  in  linear  programming. 
Further,  one  variant  of  this  method  is  theoretically  convergent 
with  only  one  matrix  factorization  throughout  the  procedure. 


